"We are protesting the total fraud of the election and the refusal to give people a chance to choose," said Gabil Guseinle, deputy leader of the Musavat opposition party, at the protest.
Guseinle demanded "the resignation of the entire government" as up to 15,000 protesters chanted "Resign!" and "Freedom!"
The opposition in this oil-rich former Soviet state is demanding fresh elections after reports of widespread irregularities in Sunday's vote and has warned the government it faces "a change of regime" unless it complies.
The rally ended peacefully as protestors left Victory Square at the scheduled time, with opposition leaders vowing to hold a new demonstration on Saturday.
The rally was the first since the disputed polls, in which President Ilham Aliyev's Yeni (New) Azerbaijan Party maintained its overwhelming grip on parliament.
Analysts said the protest would test the political strength of the oil-rich, ex-Soviet state's opposition parties, which claim they could start a peaceful uprising along the lines of Ukraine's "Orange Revolution" last year.
Demonstrators marched Wednesday under a sea of orange flags, and opposition leaders wore orange ties in overt references to the Ukrainian protest, which toppled another entrenched post-Soviet regime.
"We will use our constitutional rights to change the results of the parliamentary elections and have new elections held," said Ali Kerimli, head of the Popular Front Party.
"If the authorities do not take into account the will of the people, we will demand a change of regime and I am sure we'll get that," he said.
In a sign of unusual unity among Azerbaijan's often squabbling opposition, the New Politics coalition joined the bigger Azadliq (Freedom) alliance for the rally.
"We, like Azadliq, are pushing for the annulment of the elections," New Politics leader Eldar Namazov told AFP.
"The government is rattled," claimed Azadliq aide Murad Gassanly.
But Aliyev, who two years ago succeeded his dying father Heydar Aliyev, has so far shown little sign of compromise.
He insists not only that the elections, in which his Yeni Azerbaijan Party won 63 seats in the 125-seat parliament, were free but also that the international community was satisfied.
So far, election officials have said they will cancel the results in just two of 125 constituencies of the republic sandwiched between Russia and Iran, and do a recount in a third. Two local administration heads have been sacked.
Western governments and institutions have lined up to criticise the way Sunday's vote was conducted, giving the opposition, which won just 10 seats, important moral support.
The Yeni Azerbaijan Party announced that it would hold its own rally in Baku on Thursday.
The European Union called on all sides "to respect due process of law and to refrain from violent confrontation."